An analysis on Frequency of terms for Text Categorization

Preliminary results on a way to reduce terms for text categorization are presented. We have used the transition point; a frequency which splits the words of a text into high frequency words and low frequency words. Thresholds outcoming from document frequency of terms, Information Gain and χ were tested in combination with the transition point. A text categorization experiment based on Rocchio’s method showed that selecting terms whose frequency is lesser than the transition point discarded noise terms without diminishing the categorization task performance. In our experiment, the best result was for term selection based on document frequency of terms threshold in combination with the transition point as a cut.